Speech-centric machine learning systems have revolutionized many leading domains ranging from transportation and healthcare to education and defense, profoundly changing how people live, work, and interact with each other. However, recent studies have demonstrated that many speech-centric ML systems may need to be considered more trustworthy for broader deployment. Specifically, concerns over privacy breaches, discriminating performance, and vulnerability to adversarial attacks have all been discovered in ML research fields. In order to address the above challenges and risks, a significant number of efforts have been made to ensure these ML systems are trustworthy, especially private, safe, and fair. In this paper, we conduct the first comprehensive survey on speech-centric trustworthy ML topics related to privacy, safety, and fairness. In addition to serving as a summary report for the research community, we point out several promising future research directions to inspire the researchers who wish to explore further in this area.

语音情感识别（SER）处理语音信号以检测和表征表达的感知情绪。许多SER应用系统经常获取和传输在客户端收集的语音数据，以远程云平台进行推理和决策。然而，语音数据不仅涉及在声乐表达中传达的情绪，而且还具有其他敏感的人口特征，例如性别，年龄和语言背景。因此，塞尔系统希望能够在防止敏感和人口统计信息的意外/不正当推论的同时对情感构建进行分类的能力。联合学习（FL）是一个分布式机器学习范例，其协调客户端，以便在不共享其本地数据的情况下协同培训模型。此培训方法似乎是安全的，可以提高SER的隐私。然而，最近的作品表明，流动方法仍然容易受到重建攻击和会员推论攻击等各种隐私攻击的影响。虽然这些大部分都集中在计算机视觉应用程序上，但是使用FL技术训练的SER系统中存在这种信息泄漏。为了评估使用FL培训的SER系统的信息泄漏，我们提出了一个属性推理攻击框架，其分别涉及来自共享梯度或模型参数的客户端的敏感属性信息，分别对应于FEDSGD和FADAVG训练算法。作为一种用例，我们使用三个SER基准数据集来统一地评估我们预测客户的性别信息的方法：IEMocap，Crema-D和MSP-EXPLA。我们表明，使用FL培训的SER系统可实现属性推理攻击。我们进一步确定大多数信息泄漏可能来自SER模型中的第一层。

A major goal of multimodal research is to improve machine understanding of images and text. Tasks include image captioning, text-to-image generation, and vision-language representation learning. So far, research has focused on the relationships between images and text. For example, captioning models attempt to understand the semantics of images which are then transformed into text. An important question is: which annotation reflects best a deep understanding of image content? Similarly, given a text, what is the best image that can present the semantics of the text? In this work, we argue that the best text or caption for a given image is the text which would generate the image which is the most similar to that image. Likewise, the best image for a given text is the image that results in the caption which is best aligned with the original text. To this end, we propose a unified framework that includes both a text-to-image generative model and an image-to-text generative model. Extensive experiments validate our approach.

Monitoring water is a complex task due to its dynamic nature, added pollutants, and land build-up. The availability of high-resolu-tion data by Sentinel-2 multispectral products makes implementing remote sensing applications feasible. However, overutilizing or underutilizing multispectral bands of the product can lead to inferior performance. In this work, we compare the performances of ten out of the thirteen bands available in a Sentinel-2 product for water segmentation using eight machine learning algorithms. We find that the shortwave infrared bands (B11 and B12) are the most superior for segmenting water bodies. B11 achieves an overall accuracy of $71\%$ while B12 achieves $69\%$ across all algorithms on the test site. We also find that the Support Vector Machine (SVM) algorithm is the most favourable for single-band water segmentation. The SVM achieves an overall accuracy of $69\%$ across the tested bands over the given test site. Finally, to demonstrate the effectiveness of choosing the right amount of data, we use only B11 reflectance data to train an artificial neural network, BandNet. Even with a basic architecture, BandNet is proportionate to known architectures for semantic and water segmentation, achieving a $92.47$ mIOU on the test site. BandNet requires only a fraction of the time and resources to train and run inference, making it suitable to be deployed on web applications to run and monitor water bodies in localized regions. Our codebase is available at https://github.com/IamShubhamGupto/BandNet.

This work presents a physics-informed deep learning-based super-resolution framework to enhance the spatio-temporal resolution of the solution of time-dependent partial differential equations (PDE). Prior works on deep learning-based super-resolution models have shown promise in accelerating engineering design by reducing the computational expense of traditional numerical schemes. However, these models heavily rely on the availability of high-resolution (HR) labeled data needed during training. In this work, we propose a physics-informed deep learning-based framework to enhance the spatial and temporal resolution of coarse-scale (both in space and time) PDE solutions without requiring any HR data. The framework consists of two trainable modules independently super-resolving the PDE solution, first in spatial and then in temporal direction. The physics based losses are implemented in a novel way to ensure tight coupling between the spatio-temporally refined outputs at different times and improve framework accuracy. We analyze the capability of the developed framework by investigating its performance on an elastodynamics problem. It is observed that the proposed framework can successfully super-resolve (both in space and time) the low-resolution PDE solutions while satisfying physics-based constraints and yielding high accuracy. Furthermore, the analysis and obtained speed-up show that the proposed framework is well-suited for integration with traditional numerical methods to reduce computational complexity during engineering design.

在过去的几年中，多方计算（MPC）作为安全计算模型一直在越来越受欢迎，尤其是对于机器学习（ML）推断。与竞争对手相比，MPC的开销少于同构加密（HE），并且比基于硬件的可信执行环境（TEE）（例如Intel SGX）具有更强的威胁模型。尽管具有明显的优势，但在应用于ML算法时，MPC协议仍然与针对性相比，仍要支付大量的绩效罚款。开销是由于增加的计算和通信成本。对于在ML算法中无处不在的乘法，MPC协议在MPC服务器之间增加了32x更多的计算成本和1轮广播。此外，由于SoftMax，Relu和其他非线性操作，其具有微不足道的成本的ML计算由于增加了沟通而变得非常昂贵。这些添加的开销使MPC不太适合在实时ML推理框架（例如语音翻译）中部署。在这项工作中，我们提出了MPC-Pipe，这是一种使用两种ML特异性方法的MPC管道推理技术。 1）内线间管道和2）内层管道。这两种技术缩短了机器学习模型的总推理运行时。与当前的MPC协议实现相比，当模型权重公开时，我们的实验已显示可将ML推断潜伏期降低多达12.6％，而在模型权重公开时，将ML推断潜伏期最高12.6％。

语义3D场景理解是机器人技术至关重要的问题。尽管在空间感知方面已经取得了重大进展，但机器人仍然远非对普通人的家庭对象和位置具有常识性知识。因此，我们研究了大型语言模型来传授常识以进行场景理解。具体来说，我们介绍了三个范式，用于利用语言根据其包含的对象在室内环境中分类房间：（i）零摄像的方法，（ii）馈送前向分类器方法，以及（iii）对比分类器方法。这些方法在现代空间感知系统产生的3D场景图上运行。然后，我们分析了每种方法，证明了由于使用语言而引起的显着零拍概括和传递功能。最后，我们表明这些方法还适用于从包含房间中推断建筑标签，并在真实环境中演示我们的零弹方法。所有代码均可在https://github.com/mit-spark/llm_scene_understanding上找到。

通过脑电图信号的情绪分类取得了许多进步。但是，诸如缺乏数据和学习重要特征和模式之类的问题始终是具有在计算和预测准确性方面改进的领域。这项工作分析了基线机器学习分类器在DEAP数据集上的性能以及一种表格学习方法，该方法提供了最新的可比结果，从而利用了性能提升，这是由于其深度学习架构而无需部署重型神经网络。

因果推理提供了一种语言，以提出纯粹统计关联以外的重要介入和反事实问题。例如，在医学成像中，我们可能希望研究遗传，环境或生活方式因素对解剖表型正常和病理变异的因果关系。但是，尽管可以可靠地构建从自动图像分割中提取的3D表面网格的解剖形状模型，但缺乏计算工具来实现有关形态变化的因果推理。为了解决这个问题，我们提出了深层结构性因果形状模型（CSM），该模型利用了高质量的网格生成技术，从几何深度学习，在深层结构性因果模型的表达框架内。 CSM可以通过反事实网格产生来实现特定于受试者的预后（“如果患者大十岁，该患者的大脑结构将如何变化？”），这与大多数当前有关纯粹人口级统计形状建模的作品形成鲜明对比。我们通过许多定性和定量实验利用了3D脑结构的大数据集，证明了Pearl因果关系层次结构的所有级别CSM的能力。

最近的基于变压器的离线视频实例细分（VIS）方法取得了令人鼓舞的结果，并明显胜过在线方法。但是，它们对整个视频的依赖以及由全时空的注意力引起的巨大计算复杂性限制了它们在现实生活中的应用中，例如处理冗长的视频。在本文中，我们提出了一个基于单级变压器的高效在线VIS框架，名为InstanceFormer，该框架特别适合长期挑战性的视频。我们提出了三个新的组件来建模短期和长期依赖性和时间连贯性。首先，我们传播了对短期更改建模的先前实例的表示形式，位置和语义信息。其次，我们在解码器中提出了一种新颖的记忆交叉注意，该记忆使网络可以在某个时间窗口内研究早期实例。最后，我们采用时间对比度损失，在所有框架的实例表示中施加连贯性。记忆注意力和时间连贯性特别有益于远程依赖建模，包括诸如遮挡等挑战的情况。所提出的实例形式优于以前的在线基准方法在多个数据集上的较大边距。最重要的是，InstanceFormer超过了挑战和长数据集（例如YouTube-Vis-2021和OVIS）的离线方法。代码可从https://github.com/rajatkoner08/instanceformer获得。